Move to constants.py alongside other dicts?

Fixed in e005b7d: moved XPRO_PLATFORM_TRANSFORM into learning_resources/etl/constants.py and import it from both xpro.py and catalog_sources.py instead of maintaining two copies of the same dict.

load_run unconditionally overwrites run.prices (loaders.py:327-328) and load_prices(run, []) clears resource_prices — omitting the key doesn't help. Since these tasks upsert the same rows as the API ETL (same etl_source + readable_id), every warehouse sync will wipe displayed prices for mitxonline/xpro/mit_edx runs until the next API run restores them — a daily, user-visible flap once the beat schedule goes live. Consider making parallel validation read-only against prod rows, or adding prices to the views first.

#3565 is aimed at handling this shift in responsibilities as we cut over different data flows.

Fixed directly in e005b7d, independent of #3565 (that PR's pull-vs-push ownership guard can't actually arbitrate this: both the API-based ETL and this warehouse-pull task pass the identical etl_source value and are both "pull" from #3565's guard's point of view, so it can't tell them apart or protect one from the other — see the PR-level note on merge order).

The actual fix: _parse_runs now sets "prices": None instead of [], and loaders.load_run/load_prices (and load_instructors) now treat None as "not provided by this source, leave the existing value alone" — the same sentinel convention load_topics already uses for topics_data. An omitted key still defaults to [] (unchanged behavior for every existing caller, all of which always pass real data), so this is additive, not a behavior change for the current API-based pipelines.

The view emits cast(program_id as varchar) ("3", "4"), but the API ETL synthesizes micromasters-program-{id} (micromasters.py:126) — prod currently has micromasters-program-3/4 (verified via the public API). Passing the bare id through creates duplicate programs, and the prune step then unpublishes the real ones; the next API run reverses it. Prefix here (and use the same prefixed value for run_id, which the API ETL also prefixes — micromasters.py:137), or fix the view in ol-data-platform.

Fixed in e005b7d: transform_micromasters_program now prefixes readable_id/run_id with learning_resources.etl.micromasters.READABLE_ID_PREFIX ("micromasters-program-"), matching the API-based ETL. Added a regression test asserting the prefix is applied.

The watermark is stamped after the fetch completes, so rows modified between query execution and completion fall outside both this pull and the next incremental window (skipped until a full refresh heals them). Capture the timestamp before executing the query, or use the max last_modified seen. Not urgent while incremental is unscheduled, but worth fixing before that tier goes live.

Fixed in e005b7d: the watermark is now captured immediately before fetch_and_upsert is called, not after it returns, so a row modified mid-fetch is still covered by the next incremental window. Added a freezegun-based regression test that advances the clock inside fetch_and_upsert and asserts the stored watermark is the pre-fetch time.

_parse_bool(1) returns False. Trino returns real bools, but StarRocks (MySQL protocol) returns 1/0 — the migration this module explicitly plans for — which would silently unpublish everything. Suggest: if isinstance(value, (bool, int)): return bool(value).

Fixed in e005b7d: _parse_bool now treats bool | int as native booleans (bool(value)), so StarRocks-style 1/0 BOOLEAN columns parse correctly instead of silently becoming False. Added (1, True)/(0, False) cases to the parametrized test.

year lands in an IntegerField — "2025" coerces fine, but an empty string would raise on save. Cheap to guard here, e.g. int(row["year"]) if row.get("year") else None.

Fixed in e005b7d: year now coerces via int(row["year"]) if row.get("year") else None, so an empty string becomes None instead of raising on save. Added a regression test.

BaseWarehouseETLTask doesn't set acks_late=True. The other long-running ETL tasks in this module do (e.g. get_ocw_data, ingest_edx_run_archive, import_*_files). A worker lost mid-pull loses the work and the message is already acked. Low-impact today (daily full_refresh=True is idempotent/self-healing, and a failed incremental leaves the watermark untouched so the next run re-covers the gap), but acks_late=True on the base would match the rest of the ETL task fleet. Optional.

Fixed in e005b7d: added acks_late = True to BaseWarehouseETLTask, matching the rest of the ETL task fleet.

Distinct from the prices-wipe and readable_id issues already flagged: this transform also omits certification, certification_type, start_date/end_date, availability, and pace, all of which the API ETL sets (micromasters.py:133-149). Those are non-destructive — loaders only write keys present in the dict — but post-cutover they'd freeze at their last API-written values. Worth recording in the cutover plan: extend the view with these columns, or explicitly keep the API path as owner of them.

instructors is the exception, and it's destructive: _program_run provides no instructors key, so load_run pops the default [] and load_instructors(run, []) deletes every RunInstructorRelationship for the run (loaders.py:224). Program-run instructors will be wiped on every warehouse sync — same class of issue as prices, and this applies to the mitxonline/xpro program runs too.

The instructors-wipe half is fixed in e005b7d: _program_run now explicitly sets "instructors": None (and "prices": None), which loaders.load_run's new None-sentinel treats as "leave alone" rather than clearing — this now applies to mitxonline/xpro/micromasters program runs alike, as you flagged.

The certification/certification_type/dates/availability/pace gap is left as-is per your note — those keys are simply omitted from the transform dicts (not set to []/falsy), and upsert_course_or_program/load_program only write keys present in the dict, so they freeze at whatever the API-based ETL last set rather than getting clobbered. Non-destructive, as you said; recording it here as the known gap to close (extend the views, or keep the API path authoritative for these fields) before cutover.

If TRINO_HOST is set, maybe these old ETL scheduled tasks should not be enabled?

        "update_edx-courses-every-1-days": {
            "task": "learning_resources.tasks.get_mit_edx_data",
            "schedule": crontab(minute=0, hour=5),
        },
        "update-micromasters-programs-every-1-days": {
            "task": "learning_resources.tasks.get_micromasters_data",
            "schedule": crontab(minute=0, hour=5),  # 1:00am EST
        },
        "update-mitxonline-courses-every-6-hours": {
            "task": "learning_resources.tasks.get_mitxonline_data",
            "schedule": crontab(minute=0, hour="*/6"),
        },
        "update-xpro-courses-every-1-days": {
            "task": "learning_resources.tasks.get_xpro_data",
            "schedule": crontab(minute=0, hour=5),  # 1:00am EST
        },

Another question - @pdpinch was wondering if it might be possible to send webhooks from data-platform instead whenever there is a new or updated course/program that should be processed from these sources. And is data-platform getting this data from the source API's (like mitlearn currently does) or from the source databases?

Also, the manner in which mitxonline courses are ingested is changing right now so that B2B/invariant course runs are ingested (currently they are not) - #3543

This is currently handled via webhooks - is this meant to replace those webhooks or complement them (ie fill any missing gaps caused by webhooks that failed to be delivered or processed for whatever reason)?

The intent is to replace the webhooks, though it does create a tradeoff of frequency/timeliness. This is largely an opportunistic body of work so there may be some additional design work necessary before it all lands. If timeliness is a core requirement for OCW data then we can remove this source from the current PR